Realist evaluation - promise, problems and practicalities

Realist evaluation is an innovative, multi-method approach to evaluating the effectiveness of complex health care interventions that is having an increasing impact on the research community. Drawing on their experience doing four realist evaluations in diverse areas of healthcare, the authors offer a comprehensive overview and critique of essential theory and practice. The first paper (Realist review and realist evaluation: an introduction) introduces the main components of the approach and shows how realist review can support realist evaluation. The second paper (Concepts and methodology for realist evaluation: help or hindrance?) provides further detail on the key concepts, shows how they can be operationalised, and discusses the advantages and difficulties of using these ideas. Following these two papers introducing and illustrating the major concepts, the third paper (Realist Evaluation: a critical realist critique) takes a step back to re-consider realist evaluation in relation to its critical realist roots, asking whether it leads to evaluators abandoning the attempt to understand (and if necessary challenge) the underlying values of health care interventions and contenting themselves merely with explicating the factors that help or hinder implementation. The fourth and final paper (Data analysis and theory development in realist evaluation) plunges back into the tangled undergrowth of multiple-method data collection and shows how disparate forms of data can be synthesised for theory development, and the results presented in a form that is useful to practitioners and policy-makers.

Factors affecting the successful implementation and sustainability of the Liverpool Care Pathway for dying patients – a realist evaluation

Objectives: The Liverpool Care Pathway for the dying patient (LCP) was designed to improve end-of-life care in generalist health care settings. Controversy has led to its withdrawal in some jurisdictions. The main objective of this research was to identify the influences that facilitated or hindered successful LCP implementation.

Method: An organisational case study using realist evaluation in one health and social care trust in Northern Ireland. Two rounds of semi-structured interviews were conducted with two policy makers and twenty two participants with experience and/or involvement in management of the LCP during 2011 and 2012.

Results: Key resource inputs included facilitation with a view to maintaining LCP ‘visibility’, reducing anxiety among nurses and increasing their confidence regarding the delivery of end-of-life care; and nurse and medical education designed to increase professional self-efficacy and reduce misuse and misunderstanding of the LCP. Key enabling contexts were consistent senior management support; ongoing education and training tailored to the needs of each professional group; and an organisational cultural change in the hospital setting that encompassed end-of-life care.

Conclusion: There is a need to appreciate the organizationally complex nature of intervening to improve end-of-life care. Successful implementation of evidence-based interventions for end-of-life care requires commitment to planning, training and ongoing review that takes account of different perspectives, institutional hierarchies and relationships and the educational needs of professional disciplines. There is a need also to recognise that medical consultants require particular support in their role as gatekeepers and as a lead communication channel with patients and their relatives.

The Management of Long-Term Sickness Absence in Large Public Sector Healthcare Organisations: A Realist Evaluation Using Mixed Methods

Purpose The success of measures to reduce long-term sickness absence (LTSA) in public sector organisations is contingent on organisational context. This realist evaluation investigates how interventions interact with context to influence successful management of LTSA. Methods Multi-method case study in three Health and Social Care Trusts in Northern Ireland comprising realist literature review, semi-structured interviews (61 participants), Process-Mapping and feedback meetings (59 participants), observation of training, analysis of documents. Results Important activities included early intervention; workplace-based occupational rehabilitation; robust sickness absence policies with clear trigger points for action. Used appropriately, in a context of good interpersonal and interdepartmental communication and shared goals, these are able to increase the motivation of staff to return to work. Line managers are encouraged to take a proactive approach when senior managers provide support and accountability. Hindering factors: delayed intervention; inconsistent implementation of policy and procedure; lack of resources; organisational complexity; stakeholders misunderstanding each other’s goals and motives. Conclusions Different mechanisms have the potential to encourage common motivations for earlier return from LTSA, such as employees feeling that they have the support of their line manager to return to work and having the confidence to do so. Line managers’ proactively engage when they have confidence in the support of seniors and in their own ability to address LTSA. Fostering these motivations calls for a thoughtful, diagnostic process, taking into account the contextual factors (and whether they can be modified) and considering how a given intervention can be used to trigger the appropriate mechanisms.

The uncritical realism of realist evaluation

This article is a response to Ray Pawson’s critique of critical realism, the philosophy of science elaborated by Roy Bhaskar. I argue with Pawson’s interpretation of critical realism’s positions on both natural and social science and his charges concerning its totalizing ontology, its arrogant epistemology and its naive methodology. The differences between critical realism and realist evaluation are not as significant as Pawson contends. The main differences between the two realisms lie in their approaches to the relationship between social structures and human agency, and between facts and values. I argue that evaluation scientists need to clearly distinguish structure and agency. They should also make their values explicit. The uncritical approach of realist evaluation, combined with its underplaying of the importance of agency, leaves it open to implication in the abuses of bureaucratic instrumentalism.

Realist evaluation: an immanent critique

This paper critically analyses realist evaluation, focussing on its primary analytical concepts: mechanisms, contexts, and outcomes. Noting that nursing investigators have had difficulty in operationalizing the concepts of mechanism and context, it is argued that their confusion is at least partially the result of ambiguities, inconsistencies, and contradictions in the realist evaluation model. Problematic issues include the adoption of empiricist and idealist positions, oscillation between determinism and voluntarism, subsumption of agency under structure, and categorical confusion between context and mechanism. In relation to outcomes, it is argued that realist evaluation's adoption of the fact/value distinction prevents it from taking into account the concerns of those affected by interventions. The aim of the paper is to use these immanent critiques of realist evaluation to construct an internally consistent realist approach to evaluation that is more amenable to being operationalized by nursing researchers.

Realist Evaluation: a theory driven approach.

Realistic Evaluation assumes that all programmes implemented in practice have an underlying theory to explain how a particular intervention is meant to work. The purpose of realist evaluation is to test the theoretical propositions underpinning the implementation of a programme in order to understand how and why it works, or might not work, in certain circumstances. The first stage of the realist evaluation is to track and articulate the programme theories to determine the evidence on the ‘official conjecture’ (Pawson et al 2004 pg 16) of how the programme is suppose to work in practice. These official conjectures are then tested and refined by gathering empirical evidence to establish causal relationships between a programme and its outcome. Evaluation of the factors and interactions between factors, supporting or hindering the implementation of a programme in practice facilitate theory refinement. Theory refinement is viewed as an iterative and cyclical process undertaken to synthesise the empirical evidence and develop mid-range theories which can be generalised and applied to other programmes to improve implementation and sustainability. In this symposium an example of realist evaluation used to test and refine the theory underpinning the implementation of Early Warning Systems (EWS) is provided to clarify how this theory driven approach can be applied in practice.

How do primary health care teams learn to integrate intimate partner violence (IPV) management? A realist evaluation protocol

Background: Despite the existence of ample literature dealing, on the one hand, with the integration of innovations within health systems and team learning, and, on the other hand, with different aspects of the detection and management of intimate partner violence (IPV) within healthcare facilities, research that explores how health innovations that go beyond biomedical issues—such as IPV management—get integrated into health systems, and that focuses on healthcare teams’ learning processes is, to the best of our knowledge, very scarce if not absent. This realist evaluation protocol aims to ascertain: why, how, and under what circumstances primary healthcare teams engage (if at all) in a learning process to integrate IPV management in their practices; and why, how, and under what circumstances team learning processes lead to the development of organizational culture and values regarding IPV management, and the delivery of IPV management services. Methods: This study will be conducted in Spain using a multiple-case study design. Data will be collected from selected cases (primary healthcare teams) through different methods: individual and group interviews, routinely collected statistical data, documentary review, and observation. Cases will be purposively selected in order to enable testing the initial middle-range theory (MRT). After in-depth exploration of a limited number of cases, additional cases will be chosen for their ability to contribute to refining the emerging MRT to explain how primary healthcare learn to integrate intimate partner violence management. Discussion: Evaluations of health sector responses to IPV are scarce, and even fewer focus on why, how, and when the healthcare services integrate IPV management. There is a consensus that healthcare professionals and healthcare teams play a key role in this integration, and that training is important in order to realize changes. However, little is known about team learning of IPV management, both in terms of how to trigger such learning and how team learning is connected with changes in organizational culture and values, and in service delivery. This realist evaluation protocol aims to contribute to this knowledge by conducting this project in a country, Spain, where great endeavours have been made towards the integration of IPV management within the health system.

Mechanisms that Trigger a Good Health-Care Response to Intimate Partner Violence in Spain. Combining Realist Evaluation and Qualitative Comparative Analysis Approaches

Background. Health care professionals, especially those working in primary health-care services, can play a key role in preventing and responding to intimate partner violence. However, there are huge variations in the way health care professionals and primary health care teams respond to intimate partner violence. In this study we tested a previously developed programme theory on 15 primary health care center teams located in four different Spanish regions: Murcia, C Valenciana, Castilla-León and Cantabria. The aim was to identify the key combinations of contextual factors and mechanisms that trigger a good primary health care center team response to intimate partner violence. Methods. A multiple case-study design was used. Qualitative and quantitative information was collected from each of the 15 centers (cases). In order to handle the large amount of information without losing familiarity with each case, qualitative comparative analysis was undertaken. Conditions (context and mechanisms) and outcomes, were identified and assessed for each of the 15 cases, and solution formulae were calculated using qualitative comparative analysis software. Results. The emerging programme theory highlighted the importance of the combination of each team’s self-efficacy, perceived preparation and women-centredness in generating a good team response to intimate partner violence. The use of the protocol and accumulated experience in primary health care were the most relevant contextual/intervention conditions to trigger a good response. However in order to achieve this, they must be combined with other conditions, such as an enabling team climate, having a champion social worker and having staff with training in intimate partner violence. Conclusions. Interventions to improve primary health care teams’ response to intimate partner violence should focus on strengthening team’s self-efficacy, perceived preparation and the implementation of a woman-centred approach. The use of the protocol combined with a large working experience in primary health care, and other factors such as training, a good team climate, and having a champion social worker on the team, also played a key role. Measures to sustain such interventions and promote these contextual factors should be encouraged.

Using realist evaluation to assess primary healthcare teams’ responses to intimate partner violence in Spain

Objective: Few evaluations have assessed the factors triggering an adequate health care response to intimate partner violence. This article aimed to: 1) describe a realist evaluation carried out in Spain to ascertain why, how and under what circumstances primary health care teams respond to intimate partner violence, and 2) discuss the strengths and challenges of its application. Methods: We carried out a series of case studies in four steps. First, we developed an initial programme theory (PT1), based on interviews with managers. Second, we refined PT1 into PT2 by testing it in a primary healthcare team that was actively responding to violence. Third, we tested the refined PT2 by incorporating three other cases located in the same region. Qualitative and quantitative data were collected and thick descriptions were produced and analysed using a retroduction approach. Fourth, we analysed a total of 15 cases, and identified combinations of contextual factors and mechanisms that triggered an adequate response to violence by using qualitative comparative analysis. Results: There were several key mechanisms —the teams’ self-efficacy, perceived preparation, women-centred care—, and contextual factors —an enabling team environment and managerial style, the presence of motivated professionals, the use of the protocol and accumulated experience in primary health care—that should be considered to develop adequate primary health-care responses to violence. Conclusion: The full application of this realist evaluation was demanding, but also well suited to explore a complex intervention reflecting the situation in natural settings.

Realist evaluation of a remote hospital programme in the context of Australian health care reforms

Background: Since 2008, Australia has embarked on major healthcare reforms extending across all states and territories. Only limited evidence linking national healthcare reforms to improvement in public hospital performance exists. This evidence gap is even more pronounced in the case of remote hospital performance. This article describes a protocol retrospectively assessing a remote hospital programme to implement emergency department performance indicators, in the context of national reforms, over a period of 7 years (2008–2014). Challenges to implementing these reforms are explored.Method: Assessing the complex scenario of reform implementation requires an in-depth analysis, offered by a Realist Evaluation framework. Within this framework, a case study design is adopted to enable descriptive analysis. Interviews with key hospital stakeholders were followed by a literature review to identify a programme theory. The programme theory was articulated in the form of a preliminary context-mechanism-outcome configuration (CMOC). This theory will underlie further data collection, analysis, and interpretation. Both Realist Evaluation and case study allow flexibility in a choice of methods; both quantitative and qualitative methods will be incorporated. The thematic analysis will be employed to identify causal relationships and linkages in collected data.Discussion: Assembled data will be used to develop final CMOC patterns. The final CMOC will help in understanding the theory and mechanisms in use in the hospital.

Evaluating impact of clinical guidelines using a realist evaluation framework

RATIONALE, AIMS AND OBJECTIVES: The Remote Primary Health Care Manuals (RPHCM) project team manages the development and publication of clinical protocols and procedures for primary care clinicians practicing in remote Australia. The Central Australian Rural Practitioners Association Standard Treatment Manual, the flagship manual of the RPHCM suite, has been evaluated for accessibility and acceptability in remote clinics three times in its 20-year history. These evaluations did not consider a theory-based framework or a programme theory, resulting in some limitations with the evaluation findings. With the RPHCM having an aim of enabling evidence-based practice in remote clinics and anecdotally reported to do so, testing this empirically for the full suite is vital for both stakeholders and future editions of the RPHCM. METHODS: The project team utilized a realist evaluation framework to assess how, why and for what the RPHCM were being used by remote practitioners. A theory regarding the circumstances in which the manuals have and have not enabled evidence-based practice in the remote clinical context was tested. The project assessed this theory for all the manuals in the RPHCM suite, across government and aboriginal community-controlled clinics, in three regions of Australia. CONCLUSION: Implementing a realist evaluation framework to generate robust findings in this context has required innovation in the evaluation design and adaptation by researchers. This article captures the RPHCM team's experience in designing this evaluation.

Some reflections on Pawson’s science of evaluation: A realist manifesto

This article examines the evolution of Ray Pawson’s realist theory of evaluation, with a particular focus on his most recent book, Science of Evaluation: A Realist Manifesto. It is not a substitute for reading the original text. Reviews always say less about a book than the book in question. The goal is to provide a broader context for interpretation and an invitation to consider critically the practical import of Pawson’s grand ambitions for a new evaluation science. Like previous writings, this latest call-to-arms will appeal to some quarters of the evaluation community and dismay others. Regardless, evaluators should not remain indifferent. Pawson presents a vigorous, if at times irreverent, argument for advancing evaluation as a realist enterprise. Science of Evaluation deserves to be read and discussed widely.

Managing patient deterioration: evaluation of what works.

Background: Rapid Response Systems (RRS) have been implemented nationally and internationally to improve patient safety in hospital. However, to date the majority of the RRS research evidence has focused on measuring the effectiveness of the intervention on patient outcomes. To evaluate RRS it has been recommended that a multimodal approach is required to address the broad range of process and outcome measures required to determine the effectiveness of the RRS concept. Aim: The aim of this paper is to evaluate the official RRS programme theoretical assumptions regarding how the programme is meant to work against actual practice in order to determine what works. Methods: The research design was a multiple case study approach of four wards in two hospitals in Northern Ireland. It followed the principles of realist evaluation research which allowed empirical data to be gathered to test and refine RRS programme theory [1]. This approach used a variety of mixed methods to test the programme theories including individual and focus group interviews with a purposive sample of 75 nurses and doctors, observation of ward practices and documentary analysis. The findings from the case studies were analysed and compared within and across cases to identify what works for whom and in what circumstances. Results: The RRS programme theories were critically evaluated and compared with study findings to develop a mid-range theory to explain what works, for whom in what circumstances. The findings of what works suggests that clinical experience, established working relationships, flexible implementation of protocols, ongoing experiential learning, empowerment and pre-emptive management are key to the success of RRS implementation.  Conclusion:These findings highlight the combination of factors that can improve the implementation of RRS and in light of this evidence several recommendations are made to provide policymakers with guidance and direction for their success and sustainability.References: 1.Pawson R and Tilley N. (1997) Realistic Evaluation. Sage Publications; LondonType of submission: Concurrent session Source of funding: Sandra Ryan Fellowship funded by the School of Nursing & Midwifery, Queen’s University of Belfast